Fix ImportError by pinning tensorflow and tensorflow-text to 2.20.x - #5102
Merged
Conversation
chiajunglien
requested review from
SurbhiJainUSC,
bvandermoon,
darisoy,
igorts-git,
parambole,
richjames0 and
shralex
as code owners
September 2, 2026 09:37
There was a problem hiding this comment.
Code Review
This pull request updates several dependency versions in the TPU post-train requirements, including attempting to pin tensorflow and tensorflow-text to version 2.20.x to prevent an ImportError caused by version 2.21.0. However, because the generation script automatically converts exact pins (==) to minimum bounds (>=), the generated requirements still allow installing the problematic 2.21.0 version. To resolve this, the generation script should be updated to preserve exact pins for these specific packages.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
hengtaoguo
approved these changes
Sep 2, 2026
SurbhiJainUSC
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes
ImportError: /usr/local/lib/python3.12/site-packages/tensorflow_text/core/pybinds/libpybinds_library_common.so: undefined symbol: icudt77_datwhen running multimodal SFT tests (e.g.,test_qwen3_multimodal_sft.sh).DAG Error Log
Root Cause
bde759abd) automatically bumpedtensorflowandtensorflow-textto>=2.21.0in the post-train requirements.2.21.0oftensorflow-textintroduces a dependency on the ICU 77 library (icudt77_dat), which is currently missing from the base workload container environment.seed-envusing extreme lowest resolution (--resolution=lowest) attempted to fetchmarkdown==2.6.8(required bytensorboard). This ancient version fails to build on Python 3.12 due to the removal of theimpmodule.Solution
tensorflow==2.20.0andtensorflow-text==2.20.1insrc/dependencies/extra_deps/tpu_post_train_overrides.txtto rollback to the working versions.markdown>=3.4.1to the overrides to prevent theseed-envresolution failure on Python 3.12.src/dependencies/requirements/generated_requirements/tpu-post-train-requirements.txtlockfile.Tests
Gemma3-4b Multimodal SFT: https://cloudlogging.app.goo.gl/TgW1NRpZNEGPHMk27
Qwen3-vl-2b Multimodal SFT: https://cloudlogging.app.goo.gl/N61H2VC3UUPNMiK59
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.